Weight Sensitivity and Interdependence 1 Characterizing Network Complexity and Classification Efficiency by the Ratio of Weight Interdependence to Sensitivity

We extend previous research on digital filter structures and parameter sensitivity to the relationship between the nature of hidden-unit activation function, weight sensitivity and interdependence, and classification learning in neural networks. Weight sensitivity indicates the extent of variations in a network's output when reacting to small perturbations in its weights; whereas weight interdependence indicates the degree of co-linearity between weights. A combined measure (τ), defined as the ratio of weight interdependence to sensitivity, was examined in three feedforward networks employing different hidden-unit activation functions in the context of a non-linearly separable two-choice classification task. Simulation results show that τ reflects the complexity of hidden-unit activation function and determines the rate of learning quadratic classification boundary. Networks with more complex hidden-unit activation function evince a smaller τ and more rapid classification learning.